For the instantaneity of video background extraction in embedded visual systems, a rapid algorithm based on the Nearest Neighbor Pixel Gradient (N2PG) stability was proposed. Firstly, background initialization was conducted with one single frame, and the N2PG matrix of this frame was calculated. Secondly, several frames of the subsequent video were operated as reference image for background update, and the N2PG matrix of those frames were calculated in the same way. Then, it was judged rapidly that each pixel of the background model was static or nonstatic by calculating the subtraction between the N2PG matrix of the background image and the N2PG matrix of the reference image, referencing the threshold value of gradient stability estimated in real-time. Finally, the current background was obtained by updating or replacing each background pixel. In the simulation tests, compared with Kalman filtering method and Gaussian mixture model, only 10 to 50 frames were needed to get background in the algorithm based on N2PG, and the average speed of processing frames was increased by 36% and 75% respectively; compared to the modified Visual Background Extractor (ViBe) algorithm, the speed of updating background by using N2PG algorithm was doubled with the same required number of the video frames and the similar background quality. Experimental results show that the proposed algorithm has the advantages of strong adaptability, high speed and small storage, and the background extraction accuracy is also above 90%, it can satisfy the application of real-time embedded visual systems in natural environment.